Method and apparatus for controlling loop filtering or post filtering in block based motion compensationed video coding

ABSTRACT

Adjacent blocks are identified in an image. Coding parameters for the adjacent blocks are identified. Deblock filtering between the identified adjacent blocks is skipped if the coding parameters for the identified adjacent blocks are similar and not skipped if the coding parameters for the identified adjacent blocks are substantially different.

This application is a divisional based on prior U.S. Ser. No.09/817,701, filed Mar. 26, 2001 now U.S. Pat. No. 6,931,063.

BACKGROUND

Block based motion compensated video coding is used in many videocompression standards such as H.261, H.263, H263+, MPEG-1, MPEG-2, andH26L. The lossy compression process can create visual artifacts in thedecoded images, referred to as image artifacts. Blocking artifacts occuralong the block boundaries in an image and are caused by the coarsequantization of transform coefficients.

Image filtering techniques can be used to reduce artifacts inreconstructed images. Reconstructed images are the images produced afterbeing inverse transformed and decoded. The rule of thumb in thesetechniques is that image edges should be preserved while the rest of theimage is smoothed. Low pass filters are carefully chosen based on thecharacteristic of a particular pixel or set of pixels surrounding theimage edges.

Non-correlated image pixels that extend across image block boundariesare specifically filtered to reduce blocking artifacts. However, thisfiltering can introduce blurring artifacts into the image. If there arelittle or no blocking artifacts between adjacent blocks, then low passfiltering needlessly incorporates blurring into the image while at thesame time wasting processing resources.

The present invention addresses this and other problems associated withthe prior art.

SUMMARY OF THE INVENTION

Adjacent blocks are identified in an image. Coding parameters for theadjacent blocks are identified. Deblock filtering between the identifiedadjacent blocks is skipped if the coding parameters for the identifiedadjacent blocks are similar and not skipped if the coding parameters forthe identified adjacent blocks are substantially different.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing how deblock filtering is selectively skippedaccording to similarities between adjacent image blocks.

FIG. 2 is a diagram showing two adjacent image blocks having similarmotion vectors.

FIG. 3 is a diagram showing how transform coefficients are identifiedfor one of the image blocks.

FIG. 4 is a diagram showing how residual transform coefficients arecompared between two adjacent image blocks.

FIG. 5 is a block diagram showing how the video image is encoded anddecoded.

FIG. 6 is a block diagram showing how deblock filtering is selectivelyskipped in a codec.

FIG. 7 shows a table containing the results from selective deblockfilter skipping.

DETAILED DESCRIPTION

In conventional filtering methods, filter processing only considers asingle reconstructed image frame at a time. The motion-vectorinformation available at both the encoder and decoder is not used. Iftwo adjacent blocks share the same motion vector with respect to thesame reference image frame, (for a multiple reference frames system)there may be no significant difference between the image residuals ofeach block. The block boundary of these two adjacent blocks may havebeen filtered in the reference frame and should therefore not befiltered again for the current frame. If a deblock filter is usedwithout considering this motion-vector information, the conventionalfiltering process might filter the same boundary again and again fromframe to frame. This unnecessary filtering not only causes unnecessaryblurring but also results in extra filter computations.

FIG. 1 shows an image 12 that selectively filters block artifactsaccording to similarities between image blocks. The image 12 includesmultiple image blocks 14 that are each individually encoded before beingstored, transmitted, etc. The borders between some of the blocks 14include blocking artifacts 18. Blocking artifacts are any imagediscontinuities between blocks 14 that may be created by the encodingprocess. A low pass filter is used to reduce the blocking artifacts thatexist at the borders of adjacent image blocks.

For example, blocking artifacts 24 exist between blocks 20 and 22. A lowpass filter is used at the border 26 between blocks 20 and 22 to removeor reduce the blocking artifacts 24. The low pass filter in one exampleselects a group of pixels 28 from both sides of the border 26. Anaverage pixel value is derived from the group of pixels 28. Then eachindividual pixel is compared to the average pixel value. Any pixels ingroup 28 outside of a predetermined range of the average pixel value isthen replaced with the average pixel value.

As described above, if there are little or no blocking artifacts 24between the adjacent pixels, then the group of pixels 28 may beneedlessly filtered causing blurring in the image and wasting processingresources. A skip mode filtering scheme uses the motion estimation andcompensation information for adjacent image blocks. If the motionestimation and compensation information is similar, deblock filtering isskipped. This not only avoids unnecessary image blurring but alsosignificantly reduces the required number of filtering operations.

For example, it is determined during the encoding process that adjacentimage blocks 30 and 32 have similar coding parameters. Accordingly,deblock filtering is skipped for the groups of pixels 34 that extendacross the border 31 between adjacent blocks 30 and 32. Skip modefiltering can be used for any horizontal or vertical boundary betweenany adjacent blocks in image 12.

FIG. 2 shows reference frames 42 and 48 and a current frame 40 that iscurrently being encoded or decoded. Coding parameters for blocks 44 and46 are compared to determine whether deblock filtering should be skippedbetween the two adjacent blocks 44 and 46. One encoding parameter thatis compared is the Motion Vectors (MV) for the blocks 44 and 46.

The motion vector MV1 points from block 44 in current image frame 40 toan associated block 44′ in the reference image 42. The motion vector MV2points from block 46 in current image frame 40 to an associated block46′ in reference frame 42. Skip mode filtering checks to see if themotion vectors MV1 and MV2 point to adjacent blocks in the samereference frame 42. If the motion vectors point to adjacent blocks inthe reference frame (MV1=MV2), then deblock filtering is skipped. Thismotion vector information may be used along with other codinginformation to decide whether to skip deblock filtering between the twoimage blocks 44 and 46.

More than one reference frame may be used during the encoding anddecoding process. For example, there may be another reference frame 48.The adjacent blocks 44 and 46 may have motion vectors pointing todifferent reference frames. In one embodiment, the decision to skipdeblock filtering depends on whether the motion vectors for the twoadjacent blocks point to the same reference frame. For example, imageblock 44 may have a motion vector 49 pointing to reference frame 48 andimage block 46 may have the motion vector MV2 pointing to referenceframe 42. Deblock filtering is not skipped in this example because themotion vectors 49 and MV2 point to different reference frames.

FIG. 3 shows another coding parameter that may be used to decide whetheror not to skip deblock filtering. The image block 44 from image frame 40is compared with reference block 44′ from the reference frame 42 pointedto by the motion vector MV1 as previously shown in FIG. 2. A residualblock 44″ is output from the comparison between image block 44 andreference block 44′. A transform 50 is performed on the residual block44″ creating a transformed block 44′″ of transform coefficients. In oneexample, the transform 50 is a Discrete Cosine Transform. Thetransformed block 44′″ includes a D.C. component 52 and A.C. components53.

The D.C. component 52 refers to a lowest frequency transform coefficientin image block 44. For example, the coefficient that represents theaverage energy in the image block 44. The A.C. components 53 refer tothe transform coefficients that represent the higher frequencycomponents in the image block 44. For example, the transformcoefficients that represent the large energy differences between pixelsin the image block 44.

FIG. 4 shows the transformed residual blocks 44′″ and 46′″. The D.C.components 52 from the two transformed blocks 44′″ and 46′″ are comparedin processor 54. If the D.C. components are the same or within somerange of each other, the processor 54 notifies a deblock filteroperation 56 to skip deblock filtering between the border of the twoadjacent blocks 44 and 46. If the D.C. components 52 are not similar,then no skip notification is initiated and the border between blocks 44and 46 is deblock filtered.

In one embodiment, skip mode filtering is incorporated into theTelecommunications Sector of the International Telecommunication Union(ITU-T) proposed H.26L encoding scheme. The H.26L scheme only uses 4×4integer Discrete Cosine Transform (DCT) blocks. Here, only the D.C.component of the two adjacent blocks may be checked. However somelimited low frequency A.C. coefficients could also be checked when theimage blocks are bigger sizes, such as 8×8 or 16×16 blocks. For example,the upper D.C. component 52 and the three lower frequency A.C. transformcoefficients 53 for block 44′″ may be compared with the upper D.C.component 52 and three lower frequency A.C. transform coefficients 53for block 46′″. Different combinations of D.C. and/or low frequency A.C.transform coefficients can be used to identify the relative similaritybetween the two adjacent blocks 44 and 46.

The processor 54 can also receive other coding parameters 55 that aregenerated during the coding process. These coding parameters include themotion vectors and reference frame information for the adjacent blocks44 and 46 as described above. The processor 54 uses all of these codingparameters to determine whether or not to skip deblock filtering betweenadjacent image blocks 44 and 46. Other encoding and transform functionsperformed on the image may be carried out in the same processor 54 or ina different processing circuit. In the case where all or most of thecoding is done in the same processor, the skip mode is simply enabled bysetting a skip parameter in the filtering routine.

FIG. 5 shows how skip mode filtering is used in a block-basedmotion-compensated Coder-Decoder (Codec) 60. The codec 60 is used forinter-frame coding. An input video block from the current frame is fedfrom box 62 into a comparator 64. The output of a frame buffering box 80generates a reference block 81 according to the estimated motion vector(and possible reference frame number). The difference between the inputvideo block and the reference block 81 is transformed in box 66 and thenquantized in box 68. The quantized transform block is encoded by aVariable Length Coder (VLC) in box 70 and then transmitted, stored, etc.

The encoding section of the codec 60 reconstructs the transformed andquantized image by first Inverse Quantizing (IQ) the transformed imagein box 72. The inverse quantized image is then inverse transformed inbox 74 to generate a reconstructed residual image. This reconstructedresidual block is then added in box 76 to the reference block 81 togenerate a reconstructed image block. Generally the reconstructed imageis loop filtered in box 78 to reduce blocking artifacts caused by thequantization and transform process. The filtered image is then bufferedin box 80 to form reference frames. The frame buffering in box 80 usesthe reconstructed reference frames for motion estimation andcompensation. The reference block 81 is compared to the input videoblock in comparator 64. An encoded image is output at node 71 from theencoding section and is then either stored or transmitted.

In a decoder portion of the codec 60, a variable length decoder (VLD)decodes the encoded image in box 82. The decoded image is inversequantized in box 84 and inverse transformed in box 86. The reconstructedresidual image from box 86 is added in the summing box 88 to thereference block 91 before being loop filtered in box 90 to reduceblocking artifacts and buffered in box 92 as reference frames. Thereference block 91 is generated from box 92 according to the receivedmotion vector information. The loop filtered output from box 90 canoptionally be post filtered in box 94 to further reduce image artifactsbefore being displayed as a video image in box 96. The skip modefiltering scheme can be performed in any combination of the filteringfunctions in boxes 78, 90 and 94.

The motion estimation and compensation information available duringvideo coding are used to determine when to skip deblock filtering inboxes 78, 90 and/or 94. Since these coding parameters are alreadygenerated during the encoding and decoding process, there are noadditional coding parameters that have to be generated or transmittedspecially for skip mode filtering.

FIG. 6 shows is further detail how skip mode filtering is used in thefilters 78, 90, and/or 94 in the encoder and decoder in FIG. 5. Theinterblock boundary between any two adjacent blocks “j” and “k” is firstidentified in box 100. The two blocks may be horizontally or verticallyadjacent in the image frame. Decision box 102 compares the motion vectormv(j) for block j with the motion vector mv(k) for block k. It is firstdetermined whether the two adjacent blocks j and k have the same motionvector pointing to the same reference frame. In other words, the motionvectors for the adjacent blocks point to adjacent blocks (mv(j)=mv(k))in the same reference frame (ref(j)=ref(k)).

It is then determined whether the residual coefficients for the twoadjacent blocks are similar. If there is no significant differencebetween the image residuals of the adjacent blocks, for example, the twoblocks j and k have the same of similar D.C. component (dc(j)=dc(k)),then the deblock filtering process in box 104 is skipped. Skip modefiltering then moves to the next interblock boundary in box 106 andconducts the next comparison in decision box 102. Skip mode filteringcan be performed for both horizontally adjacent blocks and verticallyadjacent blocks.

In one embodiment, only the reference frame and motion vectorinformation for the adjacent image blocks are used to determine blockskipping. In another embodiment, only the D.C. and/or A.C. residualcoefficients are used to determine block skipping. In anotherembodiment, the motion vector, reference frame and residual coefficientsare all used to determine block skipping.

The skip mode filtering scheme can be applied to spatially sub-sampledchrominance channels For example in a case with 4:2:0 color formatsequences, skip mode filtering for block boundaries may only rely on theequality of motion vectors and D.C. components for the luminancecomponent of the image. If the motion vectors and the D.C. componentsare the same, deblock filtering is skipped for both the luminance andchrominance components of the adjacent image blocks. In anotherembodiment, the motion vectors and the D.C. components are consideredseparately for each luminance and chrominance component of the adjacentblocks. In this case, a luminance or chrominance component for adjacentblocks may be deblock filtered while the other luminance or chrominancecomponents for the same adjacent blocks are not deblock filtered.

FIG. 7 includes a table 110 showing the results of skip mode filteringusing a ITU-TH.26L Testing Model-Long TML5.0. Table 110 compares theresults of the TML filtering standard with skip mode filtering asdescribed above. Encoding results using skip mode filtering are shown intable 110 under the heading SLA.

There were four images that were tested, Akiyo_cif for 300 frames at 30Frames Per Second (fps), Foreman_cif for 300 frames at 30 fps,Foreman_qcif for 100 frames at 10 fps, and Tempete_cif for 260 frames at30 fps. Quantization Parameters (QP) of 25 and 30 were used. The resultsshow no significant visual quality degradation with the skip modefiltering. The Picture Signal to Noise Ratio (PSNR) for the images staysapproximately the same for the luminance Y and chrominance U and Vchannels. However, skip mode filtering provides time savings of 40-70percent.

Skip mode filtering can be used with any system that encodes or decodesmultiple image frames. For example, DVD players, video recorders, or anysystem that transmits image data over a communications channel, such asover television channels or over the Internet.

The skip mode filtering described above can be implemented withdedicated processor systems, micro controllers, programmable logicdevices, or microprocessors that perform some or all of the operations.Some of the operations described above may be implemented in softwareand other operations may be implemented in hardware.

For the sake of convenience, the operations are described as variousinterconnected functional blocks or distinct software modules. This isnot necessary, however, and there may be cases where these functionalblocks or modules are equivalently aggregated into a single logicdevice, program or operation with unclear boundaries. In any event, thefunctional blocks and software modules or described features can beimplemented by themselves, or in combination with other operations ineither hardware or software.

Having described and illustrated the principles of the invention in apreferred embodiment thereof, it should be apparent that the inventionmay be modified in arrangement and detail without departing from suchprinciples. Claim is made to all modifications and variation comingwithin the spirit and scope of the following claims.

1. A method for encoding an image, comprising: identifying spatiallyadjacent blocks in a same frame of the image; identifying transformcoefficients motion vectors used for motion compensation for thespatially adjacent blocks in the same frame of the image and associatedreference frames for the motion vectors of the spatially adjacentblocks; comparing the transform coefficients betweenjudging whether toperform deblock filtering for removing image residuals between theidentified spatially adjacent block caused by encoding the image, by:determining whether the motion vectors for the spatially adjacent blocksin the same frame of the image are the same or similar to each other anddetermining whether the reference frames for motion vectors of thespatially adjacent blocks are identical; and skipping deblock filteringfor removing image residuals caused by encoding the image between theidentified spatially adjacent blocks when the comparison indicates thatthe motion vectors for the spatially adjacent blocks in the same frameof the image have are determined to be the same or similar transformcoefficients; and deblock filtering to remove image residuals betweenthe identified spatially adjacent blocks when the comparison between thetransform coefficients for the identified spatially adjacent blocks inthe same frame of the image indicate that the spatially adjacent blocksdo not have the same or similar transform coefficients and the referenceframes for the motion vectors of the spatially adjacent blocks aredetermined to be identical.
 2. A method according to claim 1 including:identifying D.C. components in the transform coefficients for thespatially adjacent blocks; comparing the D.C. components for thespatially adjacent blocks with each other; and skipping deblockfiltering between the spatially adjacent blocks when the D.C. componentsfor the compared spatially adjacent blocks have same or similar valuesrelative to each other.
 3. A method according to claim 1 including:identifying both D.C. and A.C. components in the transform coefficientsfor the spatially adjacent blocks; comparing the D.C. components foradjacent blocks and A.C. components for adjacent blocks; and skippingdeblock filtering between the adjacent blocks when both the D.C.components are the same or similar and the A.C. components are the sameor similar.
 4. A method according to claim 1 including controllingdeblock filtering for a loop filter in an image coder.
 5. A methodaccording to claim 1 including controlling deblock filtering in one orboth of a loop filter and a post filter in an image decoder.
 6. A methodaccording to claim 1 including: identifying similarities between codingparameters in a luminance channel of the adjacent blocks; andcontrolling deblock filtering for both the luminance channel and achrominance channel in the image according to identified similarities inthe luminance channel.
 7. A method according to claim 1 includingselectively skipping deblock filtering in any one of a H.261, H.263,H263+, MPEG-1, MPEG-2, or H26L encoding standard according to encodingparameter similarities between adjacent image blocks.
 8. An encoder forencoding an image, comprising: a processor adapted to: compare blocks ina same image frame with reference frames; transform a result of thecomparison between the reference frames and the blocks in the imageframe into transformed blocks having transform coefficients;identifymotion vectors and associated reference frames for the motion vectors ofspatially adjacent blocks in a same frame of the image; compare are thesimilarities between the transform coefficientsdetermine whether motionvectors for spatially adjacent transformed blocks within the same imageframe are the same or similar to each other and determine whetherreference frames for the motion vectors of the spatially adjacent blocksare identical; and skippingskip deblock filtering between spatiallyadjacent transformed blocks in the image that have the same or similartransform coefficients motion vectors and that have identical referenceframes for the motion vectors.
 9. An encoder according to claim 8wherein the processor is adapted to identify motion vectors andassociated reference frames for the adjacent blocks and skip deblockfiltering between the spatially adjacent blocks according to theidentified motion vectors and reference frames.
 10. An encoder accordingto claim 8 wherein the processor is adapted to identify residualcoefficients for the spatially adjacent blocks and skip deblockfiltering according to the identified residual coefficients.
 11. Anencoder according to claim 8 wherein the processor is adapted to skipdeblock filtering according to both D.C. and A.C. components in theresidual coefficients.
 12. A decoder for decoding an encoded image,comprising: a processor adapted to identify spatially adjacent blocks inthe encoded image; identify coding parameters including D.C.coefficients motion vectors and associated reference frames for themotion vectors of the spatially adjacent blocks within a same encodedimage frame; compare the motion vectors and associated references framesfor the motion vectors of the identified D.C. coefficients betweencoding parameters of the identified spatially adjacent blocks; andenable or disable filtering of blocking artifacts between the spatiallyadjacent blocks according to the comparison of the D.C. coefficientscoding parameters between the spatially adjacent blocks within the sameencoded image frame, wherein the processor skips deblock filtering whenthe motion vectors for the spatially adjacent blocks are the same orsimilar and the reference frames for motion vectors of the spatiallyadjacent blocks are identical.
 13. A decoder according to claim 12wherein the processor is adapted to identify both D.C. coefficients andA.C. coefficients for the spatially adjacent blocks within a sameencoded image frame; compare the identified D.C. coefficients betweenthe identified spatially adjacent blocks and corn are compare theidentified A.C. coefficients between the identified spatially adjacentblocks; and enable or disable filtering of blocking artifacts betweenthe spatially adjacent blocks according to the comparison of the D.C.coefficients between the adjacent blocks within the same encoded imageframe and according to the comparison of the A.C. coefficients betweenthe adjacent blocks within the same encoded image frame.
 14. A decoderaccording to claim 12 wherein the processor is adapted to identifyresidual coefficients for the spatially adjacent blocks and skip deblockfiltering between the spatially adjacent blocks according to theidentified residual coefficients.
 15. A decoder according to claim 12wherein the processor is adapted to inverse transform the encoded image,compare blocks in the inverse transformed encoded image with referenceframes, generate a reconstructed image from the comparison between theinverse transformed encoded image and the reference frame, and skipdeblock filtering between spatially adjacent blocks in the reconstructedimage according to the coding parameters for the spatially adjacentblocks.
 16. A decoder according to claim 15 wherein the codingparameters include transform coefficients, motion vectors, and referenceframe information and the processor skips deblock filtering when thetransform coefficients, motion vectors, and reference frame informationfor the spatially adjacent blocks are the same or similar.
 17. A decoderaccording to claim 12 wherein the processor is adapted to skip deblockfiltering in one or both of a loop filter and a post filter.